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Abstract.  We  propose  a  wearable  system  that  helps  visually  impaired  persons 
walk  to  their  destination.  Once  we  choose  a  destination,  our  system  computes  a 
path  and  guides  us  using  marker  data  detected  by  the  camera  in  buildings.  It 
employs  positioning  data  from  GPS  receiver  outdoors.  Simultaneously,  it 
exploits  multiple  ultrasonic  sensors  to  avoid  obstacles  lying  in  the  path.  In  addi¬ 
tion,  we  propose  a  correction  algorithm  for  GPS  data  that  considers  speed  to  re¬ 
duce  the  positioning  error  and  deploy  a  map-  matching  algorithm  when  a  user 
breaks  way  from  the  correct  path.  We  evaluate  complex  structure  in  front  of  the 
user  with  patterns  and  determine  an  avoidance  direction  by  analyzing  these  pat¬ 
terns. 
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1  Introduction 

Most  visually  impaired  persons  have  difficulty  recognizing  their  current  position  and 
direction.  Therefore,  supplementary  positioning  and  guidance  services  are  very  im¬ 
portant.  These  services  involve  updating  one’s  position  and  orientation,  while  they  are 
traveling  an  intended  route;  when  they  become  lost;  they  reorient  and  reestablish  a 
route  to  the  destination.  In  last  decades,  a  variety  of  portable  or  wearable  navigation 
systems  have  been  developed  to  assist  visually  impaired  persons  during  navigation 
indoors  or  outdoors,  but  few  can  provide  dynamic  interactions  and  adaptability  to 
changes.  Systems  for  visually  impaired  persons  can  be  classified  into  three  main  cate¬ 
gories:  ETA  (Electronic  Travel  Aids),  EOA  (Electronic  Orientation  Aids),  and  PLD 
(Position  Locator  Devices)  [1].  ETA  is  a  device  that  transforms  spatial  information  of 
the  environment  conveyed  through  another  sensory  modality.  EOA  is  a  device  that 
provides  orientation  prior  to,  or  during  travel.  It  can  be  external  to  the  user,  or  can  be 
carried  by  the  user.  PLD  is  a  device  that  includes  GPS  receiver,  European  Geostatio¬ 
nary  Navigation  Overlay  Service  (EGNOS),  and  so  on  [1]. 

Our  system  is  categorized  as  ETA.  It  enables  us  to  walk  safely  to  our  destination 
because  ultrasonic  sensors  scan  the  complex  scene  and  an  embedded  camera  detects 
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the  markers  to  compute  the  direction  to  destination  indoors.  In  addition,  it  provides 
path-finding  features,  as  localization  and  navigation  using  GPS  and  magnetic  compass 
sensor  outdoors.  We  propose  an  optimal  layout  of  ultrasonic  sensors  that  uses  a  min¬ 
imum  number  of  sensors  covering  a  maximum  range. 

The  rest  of  the  paper  is  structured  as  follows.  We  will  give  a  brief  review  of  related 
work  in  section  2,  and  explain  our  system  in  section  3.  Experimental  results  are  pre¬ 
sented  in  section  4.  Finally,  we  conclude  our  work  in  section  5. 

2  Related  Work 

ETA  can  be  implemented  as  a  portable  system.  It  uses  ultrasonic  or  laser  sensors  to 
detect  obstacles,  and  exploits  a  CCD  camera  to  acquire  images  and  to  find  a  pathway 
with  image  processing  techniques.  The  representative  ETA  systems  are  GuideCane 
[2],  CyARM  [3]  and  so  on  [4,  5].  However,  they  require  heavy  computation,  and  take 
much  time  to  get  environmental  information  surrounding  users  such  as  distance, 
direction,  and  height  of  objects.  In  addition,  they  require  much  training.  Another  me¬ 
thod  is  to  use  various  sensors  of  ETA  that  combines  range  data  with  information  ac¬ 
quired  from  mobile  robots.  MELDOG  [6]  Walking  Guide  Robot  [7],  and  so  on  [8,  9] 
are  categorized  as  EOA  systems.  However,  avoidance  direction  changes  frequently, 
since  they  do  not  consider  that  the  human  body  trembles  while  walking.  Also,  the 
robots  can  only  support  horizontal  movement  and  its  mobility  is  much  more  restricted 
than  is  the  case  of  wearable  systems. 

We  are  mostly  interested  in  ETA  that  includes  more  specifically  obstacle  detection 
systems,  in  addition  to  guidance  systems.  These  can  be  categorized  depending  on  how 
the  information  is  gathered  and  how  this  information  is  given  to  the  user.  The  Nation¬ 
al  Research  Council’s  guidelines  for  ETA  are  listed  [10].  We  designed  our  system 
based  on  these  requirements. 

3  Wearable  Computer  System  Considering  Spatial  Context 

In  this  section,  we  explain  our  system  that  guides  the  user  to  the  destination  with 
marker  position  and  orientation  in  a  building.  In  addition,  our  system  helps  a  user  to 
arrive  at  the  destination  using  positioning  data  from  GPS  receiver  and  magnetic  com¬ 
pass  sensor  outdoors.  It  uses  an  ultrasonic  sensor  array  to  detect  and  avoid  obstacles. 

We  gather  data  from  an  ultrasonic  sensor  array,  marker  camera,  GPS  receiver,  and 
magnetic  compass  sensor.  They  are  transmitted  to  a  wearable  computer.  As  depicted 
in  Fig.  1,  the  indoor  guidance  system  is  composed  of  two  modules:  marker  recogni¬ 
tion  and  obstacle  avoidance  (grey).  The  right  column  of  Fig.  1  represents  the  flow¬ 
chart  of  the  outdoors  navigation  system  composed  of  positioning  and  tracking  module 
and  obstacle  detection  module  (grey).  These  steps  are  detailed  in  the  next  subsections. 

3.1  Obstacle  Detection  Technique  Using  Ultrasonic  Sensor  Array 

One  of  the  important  things  to  recognize  forward  space  with  range  sensors  is  to  ar¬ 
range  the  sensors  efficiently  to  cover  a  maximum  range  with  a  minimum  number  of 
sensors  since  a  sensor  has  physical  characteristics  such  as  coverage  and  maximum 
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Fig.  l.  Procedure  for  safe  guidance  indoors  (left)  and  outdoors  (right) 


detection  range.  The  sensor  array  should  be  designed  not  to  be  influenced  by  the  mo¬ 
tion  of  the  arm  or  upper  body  while  walking.  In  addition,  it  is  important  to  make  the 
sensor  array  detect  obstacles  quickly  and  precisely. 


Fig.  2.  Example  layout  of  ultrasonic  sensors  and  geometry  covering  forward  space 

Each  sensor  has  a  direction  and  coverage  of  a  conic  shape  that  does  not  overlap 
those  of  neighboring  sensors  and  acquires  range  data  for  its  direction,  as  shown  in 
Fig.  2.  Eight  sensors  are  arranged  in  four  rows.  One  row  is  made  up  two  sensors.  Sen¬ 
sors  of  the  top  row  detect  objects  hazardous  to  the  user’s  head,  those  of  the  second 
and  third  row  detect  frontal  obstacles,  such  as  a  wall  or  furniture,  and  those  of  the 
bottom  row  recognize  small  obstacles  on  the  floor.  We  evaluate  Hf  of  the  height  of 
frame,  where  the  user’s  height  is  H  and  the  sensing  range  of  an  ultrasonic  sensor  is 
D range-  &  denotes  the  beam  angle  of  each  sensor  and  y  indicates  a  tilting  angle  thre¬ 
shold  by  changing  Hf ,  as  shown  in  equation  1.  The  target  area  is  defined  as  an  area  of 
any  shape  in  the  2D  plane  to  cover  the  sensor  array,  as  described  in  Fig.  3.  This  guar¬ 
antees  high-speed  and  accurate  range  sensing,  since  it  scans  a  scene  immediately,  as 
depicted  in  Fig.  3  (in  the  case  of  the  sensing  range  is  Drange). 
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Fig.  3.  Scanning  coverage  of  sensors  in  our  system  (Scan  range  of  ultrasonic  sensor  is  Drange.) 


D m„ge x sin + (20°  +  y)}<H  f  < Drange x [sin + (20°  +  y)}+ sin {^+(10°+^)}]  (D 

The  next  step  is  to  determine  the  optimal  direction  of  avoidance  by  analyzing  the 
range  data.  Previous  methods  for  path  generation  have  an  issue  in  demanding  much 
computation  to  generate  local  and  global  maps  in  the  preprocessing  step.  A  navigation 
method  should  be  simple  to  make  a  decision  of  optimal  direction  in  real  time  with  a 
small  amount  of  computation  to  assist  a  user’s  walk-through  on  a  low  performance 
mobile  computing  device.  The  four  sensors  of  the  second  row  and  third  row  detect 
obstacles  in  front  of  the  user.  We  consider  the  range  data  acquired  from  them  and 
represent  spatial  information  as  patterns  of  the  range  data.  The  range  data  are  classi¬ 
fied  into  four  cases:  danger  (less  than  100  cm),  warning  (100-130  cm),  adequacy 
(130-200  cm),  and  unconcern  (more  than  200  cm).  We  can  define  256  (=44)  cases 
and  form  the  corresponding  input  range  data  from  the  four  sensors  into  256  cases. 

3.2  A  Target  Guidance  Using  Optical  Marker 

We  attach  makers  to  the  ceiling  to  identify  the  current  position  and  orientation.  Our 
system  selects  the  direction  to  the  destination  by  recognizing  the  identifier  and  direc¬ 
tion  of  the  marker  with  the  camera.  We  can  estimate  the  relative  orientation  by  calcu¬ 
lating  the  direction  vector  from  the  current  position  to  the  destination.  The  camera 
detects  markers  as  a  regular  square,  and  finds  feature  points.  The  makers  comprises 
12X12  blocks;  they  can  represent  256  cases.  The  markers  are  detected  by  the  camera 
and  their  patterns  are  transmitted  to  the  wearable  computer.  The  values  are  interpreted 
and  compared  to  the  predefined  marker’s  ID.  A  marker  is  composed  of  a  unique  de¬ 
sign  and  white  patterns  on  a  black  background  to  reduce  the  load  processing  image 
[11]. 

We  determine  the  walking  direction  by  estimating  the  orientation  vectors  of  the  us¬ 
er,  detected  marker  ID  and  orientation.  We  have  to  consider  the  following  issues  in 
this  system.  We  determine  the  correspondence  between  the  attached  position  of  the 
marker  and  the  previously  stored  local  map  of  rooms.  In  addition,  it  is  important  to 
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decide  the  marker  size,  since  the  camera  should  hold  at  least  one  of  the  marker  in  its 
field  of  view  at  any  time.  We  use  multiple  markers  with  the  same  ID  for  fast  detection. 
However,  even  though  we  use  several  markers,  the  camera  may  not  detect  markers  in 
some  circumstances.  We  employ  a  method  using  the  direction  of  the  previously  de¬ 
tected  marker  to  resolve  the  problem.  We  estimate  the  orientation  vector  for  the  user 
as  the  sum  of  orientation  vector  of  the  destination  marker  detected  in  the  previous 
frame  and  the  user’s  avoidance  direction  vector.  We  regard  this  vector  as  the  decided 
orientation  until  the  camera  detects  a  new  marker.  We  keep  the  current  direction  as 
the  dot  product  of  marker  orientation  vector  and  previous  direction  vector  less  than 
pre-defined  threshold.  Therefore,  we  can  reduce  errors  due  to  a  user’s  trembling.  We 
evaluate  the  angle  using  the  dot  product  value  of  these  vectors. 

3.3  Outdoor  Navigation  System  Incorporating  Sensors 

We  can  recognize  the  position  and  orientation  of  the  user  as  data  gathered  from  a  GPS 
receiver  and  magnetic  compass  sensor  outdoors.  The  user  selects  a  destination  that  is 
one  of  the  stored  POI  (Point  Of  Interests)  lists  in  a  map  in  advance.  Then,  the  system 
guides  the  user  to  the  destination  using  the  A*  algorithm  available  in  searching  op¬ 
timal  path  [12].  We  get  longitude  and  latitude,  as  well  as  walking  speed  and  heading 
degree  from  NMEA0183  protocol  data  of  GPS  receiver.  However,  we  use  a  magnetic 
compass  sensor  simultaneously  to  acquire  the  user  orientation,  since  the  received 
heading  values  are  inaccurate  in  many  cases.  We  can  reduce  GPS  position  error  by 
considering  the  walking  speed  of  our  user.  Then,  we  assume  4.0  km/h  as  the  speed 
outdoors  through  repeated  experiment.  We  presume  the  error  value  as  1  m/sec  when 
threshold  is  more  than  the  difference  between  the  current  received  position  and  pre¬ 
vious  value.  In  addition,  our  system  computes  the  average  of  the  previous  position 
values,  including  current  position,  and  it  finds  the  nearest  position  of  the  user  in  the 
stored  POI  list.  We  have  to  predefine  the  bounds  when  we  map  for  a  visually  im¬ 
paired  person. 

The  next  step  is  to  search  an  optimal  path  for  a  visually  impaired  person  using  the 
A  algorithm  that  is  a  widely  known  path-finding  method.  Our  system  loads  the  stored 
landmarks  and  POIs,  and  informs  the  user  via  voice  messages.  The  user  selects  a  destina¬ 
tion  in  the  POI  list.  In  addition,  wherever  GPS  data  is  not  received  when  a  user  is  located 
in  narrow  alleys,  we  employ  a  concept  of  satellite-based  navigation  combined  with  dead 
reckoning.  This  is  a  guidance  method  of  determining  walking  direction  with  current  di¬ 
rection  data  of  the  magnetic  compass  sensor  and  the  previous  received  GPS  position  data. 

We  made  the  prototype  of  system.  The  main  processing  of  our  system  is  performed 
on  an  Xhyper-PX270  based  embedded  system  equipped  with  RS-232C  Bluetooth 
sensor  controller  and  vibrator  controller.  The  input  device  is  a  keypad  with  Braille 
type  buttons.  The  output  is  represented  as  a  series  of  voice  messages  or  vibration  pat¬ 
terns  through  an  earphone  and  vibration  jacket.  In  addition,  marker  information  and 
GPS  data  are  processed  in  an  embedded  ARM  10  processor.  Fig.  4  shows  our  system 
prototype.  We  made  a  sensor  frame  using  hard  leather  to  fix  the  ultrasonic  sensors  and 
to  minimize  motion  or  distortion  between  sensors.  In  addition,  we  attached  vibrators 
into  the  jacket  for  the  conveniences  of  the  user’s  perception.  We  generated  a  camera 
frame  using  hard  leather  to  fix  the  camera  on  the  shoulder  to  detect  the  marker.  Fur¬ 
thermore,  we  developed  a  cross-bag  for  the  user’s  convenience,  simultaneously. 
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Fig.  4.  System  prototype 


4  Experimental  Results 

Based  on  the  layout,  we  experimented  with  differing  heights  of  obstacles  to  confirm 
the  ultrasonic  sensor  arrangement  and  effectiveness.  Fig.  5  shows  the  results  of  detec¬ 
tion  rate  by  obstacle  height  and  the  user’s  walking  speed.  The  sensors  of  the  top  row 
detected  more  than  90%  of  hazards  to  the  user’s  head  and  the  sensors  of  the  bottom 
row  recognized  more  than  80%  of  small  obstacles  (about  10cm  high)  on  the  floor.  In 
addition,  the  sensors  of  the  second  and  third  row  detected  more  than  95%  of  the  ob¬ 
stacles  in  front  of  the  user.  However,  the  sensors  did  not  detect  obstacles  when  the 
user  walks  more  than  1.3  times  faster  than  ordinary  pace.  A  visually  impaired  person 
can  safely  walk  in  places  where  there  are  established  obstacles,  such  as  a  notice  board 
and  a  signboard,  even  narrow  hallway,  according  to  the  experimental  results. 
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Fig.  5.  The  results  for  detection  rate  of  the  top  row  sensors  (left),  bottom  row  sensors  (middle), 
and  middle  rows  of  sensors  (right) 
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Second,  we  experimented  with  different- sized  markers  to  determine  the  optimal 
size,  while  maximizing  the  recognition  rate  of  a  camera.  Fig.  6  describes  the  experi¬ 
mental  environment  and  shows  the  results.  In  Fig.  6,  the  s-axis  represents  marker  size 
and  h- axis  denotes  the  distance  from  camera  to  marker.  We  can  usually  determine  the 
optimal  marker  size  is  20  cmX20  cm,  since  the  height  from  ground  to  ceiling  is  within 
350  cm  and  the  camera  view  angle  is  45°.  We  use  multiple  markers  with  equal  IDs 
and  the  interval  between  markers  is  about  10-15  cm.  These  are  considered  to  be  the 
camera  view  angle  and  walking  speed  of  a  handicapped  user.  The  result  showed  a 
user  arrived  at  the  destination  safely  at  3  km/h. 


Fig.  6.  Experimental  environment  (left)  and  results  depending  on  marker  size  (right) 


Finally,  we  tested  the  performance  of  our  system  in  an  outdoor  environment. 
Several  trials  were  conducted  in  different  operating  conditions  to  investigate  the  per¬ 
formance  of  each  module  of  the  system.  The  paths  are  classified  into  four  cases,  ac¬ 
cording  to  the  width  and  complexity  of  the  road:  narrow  road  (less  than  3  m),  wide 
road  (more  than  3  m),  simple  path  (less  than  three  intersections),  and  congested  path 
(more  than  three  intersections).  We  performed  four  experiments:  only  using  ultrasonic 
sensor  array,  exploiting  ultrasonic  sensors  and  GPS  receiver,  applying  our  system 
with  ultrasonic  sensors,  GPS  and  compass  sensor,  and  lastly  normal  vision  at  3  km/h. 
The  our  system  achieved  a  high  success  rate  of  about  53%  if  they  walked  along  a  path 
200  m  long  on  a  wide  and  simple  road.  Therefore,  we  can  improve  the  success  rate  by 
re-searching  for  a  new  path  every  200  m,  in  the  case  of  a  long  path.  The  proposed 
system  of  an  equipped  GPS  receiver  and  magnetic  compass  sensor  improved  the 
walking  speed  from  a  minimum  18%  to  maximum  of  28%  compared  to  the  previous 
system  using  only  a  GPS  receiver,  and  success  rate  of  55%.  In  addition,  the  user  safe¬ 
ly  walks  to  the  destination  at  1.5  km/h. 


5  Conclusions 

The  main  role  of  the  guidance  system  is  to  quickly  capture  environmental  data  from 
various  sensors  and  map  the  extracted  and  processed  content  onto  available  user  inter¬ 
faces  in  the  most  appropriate  manner.  We  implemented  the  guidance  system  for  a 
visually  impaired  person.  Once  the  user  chooses  a  destination,  our  system  computes  a 
path  and  guides  the  user  using  marker  data  detected  by  the  camera  in  buildings  and 
employing  position  data  from  a  GPS  receiver  outdoors.  Furthermore,  it  exploits  mul¬ 
tiple  ultrasonic  sensors  to  avoid  obstacles  lying  in  the  path.  Walking  directions  are 
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transferred  to  the  user  as  voice  messages  or  vibration  patterns.  Therefore,  our  system 
helps  users  to  arrive  at  their  destination  safely  without  help  from  others.  Our  future 
work  will  focus  on  implementing  a  more  flexible  system  that  utilizes  the  advantages 
of  wearable  technologies  and  easily  customized  to  those. 
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